Contributing Authors
Kalina Bontcheva
University of Sheffield
Symeon Papadopoulous
Centre for Research and Technology, Hellas
Filareti Tsalakanidou
Riccardo Gallotti
Fondazione Bruno Kessler
Lidia Dutkiewicz
KU Leuven Centre for IT & IP Law - imec
Noémie Krack
Denis Teyssou
Agent France-Presse
Francesco Severio Nucci
Engineering
Jochen Spangenberg
Deutsche Welle Research and Cooperation Projects
Ivan Srba
Kempelen Institute of Intelligent Technologies
Patrick Aichroth
Fraunhofer Institute for Digital Media Technology
Luca Cuccovillo
Luisa Verdoliva
University Federico II of Naples
Generative AI and
Disinformation: Recent
Advances, Challenges, and
Opportunities
Editor
Kalina Bontcheva
University of Sheffield
Acknowledgements
This work has been co-funded by the European Union under the Horizon Europe vera.ai
project, Grant Agreement number 101070093, the UK’s innovation agency (Innovate UK)
grant 10039055, and the Swiss State Secretariat for Education, Research and Innovation
(SERI) under contract number 22.00245; TITAN project, Grant Agreement number
101070658; AI4Media project, Grant Agreement 951911; and AI4Trust project, Grant
Agreement 101070190.
Proofreading and formatting Joanna Wright, University of Sheffield.
Published February 2024
This white paper is made available in Open Access under a Creative Commons Attribution-
Non Commercial-No Derivatives 4.0 International Licence (CC BY-NC-ND)
1
Contents
1. Introduction .................................................................................................................. 3
2. Synthetically Generated Disinformation: Kinds, Prevalence, and Impact on Elections 4
2.1 AI-generated images and videos ............................................................................... 4
2.2 AI-generated audio .................................................................................................... 6
2.3 AI-generated text ....................................................................................................... 8
2.4 Threats Posed by Generative AI to 2024 Elections in Europe: An Important Case
Study ................................................................................................................................ 9
3. Recent Advances in Detection of Synthetically Generated Disinformation ................ 11
3.1 Detecting Synthetic and Manipulated Images and Videos ...................................... 11
3.2 Detecting audio deepfakes ...................................................................................... 13
3.3 Detecting machine-generated textual disinformation ............................................... 15
4. Selected AI-powered Tools for Assisting Verification Professionals and Citizens ..... 16
4.1 InVID-WeVerify plugin ............................................................................................. 16
4.2 AI-based services for coaching media professionals ............................................... 18
4.3 AI-Based services for coaching citizens .................................................................. 19
4.4 Enhancing the transparency of AI services for content verification ......................... 20
5. Ethical and Legal Issues at the Intersection of Disinformation and Generative AI ..... 21
5.1 Data Pollution, Copyright Concerns, and Power Imbalance .................................... 21
5.2 Disinformation Risks Arising from Generative AI Usage ......................................... 22
6. Challenges Ahead ...................................................................................................... 23
6.1 Generative AI, Hallucinations, and Quality of LLM Training Data ............................ 23
6.2 Overcoming Citizens’ Ill-Founded Trust in AI .......................................................... 24
6.3 Development of New Tools for Detecting AI-Generated Content ............................ 24
6.4 Beyond English: New Multilingual Detection Tools Are Needed ............................. 25
6.5 Access to Data for Researchers .............................................................................. 25
6.6 Scarcity of Research Funding .................................................................................. 26
7. Opportunities and Next Steps .................................................................................... 26
Bibliography .......................................................................................................................... 29
2
Figures
Figure 1: Evolution of fully synthetic image quality over the years. Image originally tweeted by
Ian Goodfellow (creator of GANs), then extended by (Masood et al., 2023) and then extended
again. ...................................................................................................................................... 5
Figure 2: Examples of reenactment, replacement, editing, and synthesis deepfakes of the
human face. Image by (Mirsky & Lee, 2021). ......................................................................... 6
Figure 3: Examples of fully synthetic images generated using MidJourney that became
mainstream. Articles discussing these cases: a) .................................................................... 6
Figure 4: Screenshots of the aforementioned BBC, Telegraph and Tagesschau articles (1-3)
................................................................................................................................................ 7
Figure 5: Screenshots of the aforementioned Wired and BR24 articles (4-5) ........................ 7
Figure 6: Overview of MINTIME architecture that handles the detection of deepfakes in videos
with multiple identities. Image from (Coccomini et al., 2022). ............................................... 12
Figure 7: Depictions of Fourier transform (amplitude) of the artificial fingerprint estimated from
1000 image residuals from 10 recent synthetic image generation models, which illustrates the
special artefacts introduced by generative models. Image from (Corvi et al., 2023b). ......... 13
Figure 8: Screenshot of the keyframe fragmentation service (on a fake video during the
Ukraine war debunked among others by the British fact-checker FullFact). ......................... 17
Figure 9: Screenshot of the OCR component running on a screenshot of a Telegram post of
the Kremlin chief propagandist Vladimir Soloviov on a fake video with the detection of text
blocks, the language identification and the possibility to send the block to a translation engine
in one click. ........................................................................................................................... 17
Figure 10: Screenshot of the interface of the synthetic media detector with a fake picture of
US president Joe Biden, allegedly sleeping during a G20 summit. ...................................... 18
3
1. Introduction
Over the past three years, generative AI technology (e.g. DALL-E, ChatGPT) made the
sudden leap from research papers and company labs to online services used by hundreds of
millions of people, including school children. In the United States alone, 18% of adults had
used ChatGPT according to Pew Research in July 2023 (
Park & Gelles-Watnick, 2023).
As the fluency and affordability of generative AI continues to increase from one month to the
next, so does its wide-ranging misuse for the creation of affordable, highly convincing large-
scale disinformation campaigns. Highly damaging examples of AI-generated disinformation
abound, including highly lucrative Facebook ads
1
seeking to influence voters through
deepfake videos of Moldova’s pro-Western president (Gilbert, 2024). YouTube has also been
found to host ads with political deepfake videos, which used voice imitation (RTL Lëtzebuerg,
2023). Beyond videos, AI-generated images have been used to spread disinformation about
Gaza (France, 2023; Totth, 2023) and propagate divisive, anti-immigrant narratives (The
Journal, 2023). Audio deepfakes have also been reported by fact-checkers, so far, these have
mostly focused on fake conversations and statements by politicians (Demagog, 2023;
Dobreva, 2023; Bossev, 2023). Russian disinformation campaigns have also weaponised
generative AI (e.g. a deep fake video of the Ukrainian president calling for surrender (Kinsella,
2023), an AI-generated conversation between the Ukrainian president and his wife (Demagog,
2023).
The countries being targeted span the entire European Union (and beyond), including highly
susceptible countries such as Bulgaria (
Bossev, 2023; BNT, 2023), where citizens have low
levels of media literacy and critical thinking skills, as well as a lack of awareness of the
existence of sophisticated AI-generated images, videos, audio, and text.
Platform actions aimed at countering disinformation in posts and adverts have so far also fallen
short on detecting and removing harmful AI-generated content. All major social media
platforms and chat apps have been impacted. For brevity, here we include only some
examples from Facebook (ads (
Gilbert, 2024), groups (The Journal, 2023), pages (Bossev,
2023)), YouTube (RTL Lëtzebuerg, 2023), X (France, 2023; Totth, 2023), Instagram (France,
2023; Totth, 2023), TikTok (AFP, USA & AFP Germany, 2023; Marinov, 2023) and Telegram
(Starcevic, 2023; Marinov, 2023). AI-generated content (e.g. a fake audio claiming votes are
being manipulated in Bulgaria (Dobreva, 2023)) is also being sent by email to media and
journalists, with the intention of duping reliable outlets into publishing fake content.
Moreover, not only is generative AI used to create highly deceptive disinformation campaigns
at low cost, but its existence and proficiency is being weaponised by actors who are
propagating false claims that authentic images, videos, and audio content from governments
and mainstream media are actually fake. One recent example is from a court case against
Tesla, where the company lawyers claimed that a video of Elon Musk was a deepfake (The
Guardian & Reuters, 2023). Another example is from Bulgaria where bad actors seeking to
discredit the government and the “neoliberal” mainstream media spread false claims through
1
The ads reportedly earned Meta $200,000 in revenue.
4
a pro-Kremlin Telegram channel and Facebook pages, labelling as fake an official photo of
the Bulgarian prime minister talking at the European Parliament.
What these examples demonstrate is that generative AI has had a disruptive hyper-realistic
2
effect on citizen’s ability to discern and on the platforms’ and fact-checkers’ abilities to tackle
online disinformation.
More specifically, from the perspective of verification professionals, the absence of a traceable
origin is a complete disruption to their content verification workflows. Until generative AI
became a cheap and prolific “author” of fake online content, journalists, fact-checkers, human
rights defenders and other professionals mainly relied on being able to trace a given object
(text, image, video, audio) back to its original source and thus verify whether the examined
content was consistent with reality or if, on the contrary, it came from a decontextualised or
manipulated copy.
Another particularly troubling consequence of the commodification of generative AI is in its
extremely low cost and easy accessibility through websites and mobile applications. There are
now numerous online tutorials on YouTube, TikTok, and elsewhere on how to create AI-
generated images or videos (including using AI avatars), either for free or costing as little as
tens or hundreds of dollars per month. In comparison, in 2016 the budget of Russia’s Internet
Research Agency (IRA) was $1.25 million dollars per month (
Intelligence: Senate, c. 2019 -
2021).
The goal of this white paper is to deepen understanding of the disinformation-generation
capabilities of state-of-the-art AI, as well as the use of AI in the development of new
disinformation detection technologies, along with the associated ethical and legal challenges.
We conclude by revisiting the challenges and opportunities brought by generative AI in the
context of disinformation production, spread, detection, and debunking.
2. Synthetically Generated Disinformation: Kinds,
Prevalence, and Impact on Elections
2.1 AI-generated images and videos
In terms of visual content, there have been many recent developments, often referred to as
“deepfakes”, which started from a very specific type of digital manipulation (face swapping in
particular), but is now broadly (and in most cases inaccurately) used to refer to many types of
fully synthetic or digitally manipulated images and videos (
Tolosana et al., 2020; Verdoliva
2020). The most common types of visual synthetic media include the following:
Fully synthetic images, most often depicting human faces: for several years,
Generative Adversarial Networks (GANs) and specifically the StyleGAN family of
generators (Karras et al., 2019) has been the method of choice for generating synthetic
2
The French philosopher Jean Baudrillard defined the concept of hyperreality as “the generation by
models of a real without origin or reality” (
Hyperreality, Wikipedia), while the Italian
semiotician Umberto Eco wrote about hyperreality as the “absolute fake” (Eco, U., 1967).
5
images. However, as of 2022, Diffusion Models (Rombach et al., 2022) and most
commonly publicly accessible models such as Stable Diffusion and Midjourney have
become the most popular choice for generating a variety of realistic imagery using text
prompting.
Face attribute manipulation, e.g. modifying the age of a person or adding accessories
(e.g. eye glasses, hats) on them, is another popular kind of AI manipulated media,
where both GAN and Diffusion Model architectures have evolved a lot and offer
numerous capabilities for editing an input image in terms of different attributes or for
“interpolating” between two images (e.g. starting from an image of a young boy create
intermediate versions of images that gradually end up depicting an elderly woman).
Face swapping has been one of the first kinds of manipulations that popularised the
term “deepfake” and aims to replace the face in an original image or video with a
selected one. This kind of manipulation has been maliciously used, not only for
spreading disinformation, but also in the context of image-based sexual abuse.
Face reenactment and lip synching are other common types of video manipulation that
aim at modifying the facial movements and expressions of a target person so that they
present them in a specific way, e.g. present a politician as making a specific statement,
which they did not.
There are also other kinds of synthetic media models, for instance, fully synthetic video
given a text description (the commercial service Runway ML provides such
capabilities) and another emerging area of synthetic media is based on Neural
Radiance Fields (NeRF) (
Yu et al., 2021). However, neither of those have so far
become mainstream and their use has not yet been recorded in relation to
disinformation activities.
Figure 1: Evolution of fully synthetic image quality over the years. Image originally tweeted by Ian
Goodfellow
(creator of GANs), then extended by (Masood et al., 2023) and then extended again.
6
Figure 2: Examples of reenactment, replacement, editing, and synthesis deepfakes of the human
face. Image by (
Mirsky & Lee, 2021).
Figure 3: Examples of fully synthetic images generated using MidJourney that became mainstream.
Articles discussing these cases: a) Pope jacket, b) Trump arrest, c) father carrying children in Gaza.
2.2 AI-generated audio
Similar to text and visual synthesis, AI-based speech synthesis methods have made enormous
progress in recent years, which is mainly due to the so-called end-to-end learning paradigm:
Here, text analysis, acoustic modelling and audio synthesis are no longer isolated, but
mapped, trained and optimised within a common network architecture. This network
architecture can also be trained with roughly annotated data (in the form of pairs of text and
audio recordings), which are available in almost unlimited quantities without the need for
complex expert annotation. Speech synthesis has been achieving ever better speech quality
over recent years, as early as 2017, often on par with natural speech, e.g. with approaches
such as Adobe’s VoCo, with Wavenet or Google’s Tacotron2. Since then, AI-based synthesis
methods have continuously improved, driven by commercial interests in the media industry.
These advancements are primarily focused on enhancing synthesis speed, improving control
over the output, simplifying the training process, and minimising the amount of training data
required. Recent advancements have yielded high-quality models that are accessible to non-
expert users, offering both text-to-speech and voice conversion: Text-to-speech involves
converting written text into spoken words using synthetic voice and voice conversion entails
transferring the unique characteristics of a speaker's voice from an audio recording onto
another set of spoken words. Notable examples in this area include
Resemble.AI, ElevenLabs,
7
Descript, VALL-E, each offering unique features and degrees of versatility. Furthermore, the
landscape of voice synthesis has been extended by the availability of pre-trained models of
publicly-known figures, exemplified by platforms like
FakeYou.
While speech synthesis has emerged as a powerful tool for disinformation creation, its
potential for misuse (e.g. to damage citizen trust or undermine democratic processes) has
long been underestimated. A few examples include:
1. A case reported by BBC News (Goodman & Hashim, 2023) involving the use of voice
conversion technology to impersonate a high-profile individual in the context of the
Sudan civil war
2. Fabricated Ukraine-related videos including speech synthesis, including a deepfake
video of
Volodymyr Zelensky allegedly surrendering which was reported on by The
Telegraph (March 2022)
3. German news outlet Tagesschau reported on the circulation of speech deepfakes
falsely attributed to their broadcasts (
Reveland & Siggelkow, 2023)
Figure 4: Screenshots of the aforementioned BBC, Telegraph and Tagesschau articles (1-3)
4. A deepfake audio clip of the UK politician Keir Starmer, discussed in a Wired article
(
Meaker, 2023).
5. A video of the German Chancellor Olaf Scholz involving speech synthesis, which was
covered by BR24, emphasising the seriousness and potential harm of such deepfakes,
which are often dismissed as mere pranks (BR24, 2023).
Figure 5: Screenshots of the aforementioned Wired and BR24 articles (4-5)
8
Beyond speech, generative AI can also be used to create plausible environmental noises and
sound events that simulate real-world events and make recordings appear more realistic. For
instance, Meta's Audiobox (Hsu & Akinyemi et al., 2023) demonstrates this capability by
generating unified audio with natural language prompts, showcasing the potential for such
technology in creating deceptive auditory environments.
Furthermore, AI can also be employed for singing voice synthesis, style mimicry, and
potentially imitating the overall persona of creators or artists. This can lead to the generation
of fake content (deepfakes) related to artists, misusing the emotional connection that fans
have with them.
2.3 AI-generated text
Large Language Models (LLMs) are artificial neural networks, capable of generating fluent,
highly persuasive texts and human dialogs. The most widely known and used one is OpenAI’s
ChatGPT. It is very cheap to use and is already being harnessed in disinformation campaigns
and online abuse. For instance, recent research uncovered a network of at least 64 X/Twitter
bot accounts using content generated by
ChatGPT (Thomas, 2023).
Below we include an example of artificially generated text, reinforcing a Ukraine-related false
narrative:
Prompt
Write a news article about the narrative: Ukraine hosts secret US bio-labs. Use arguments and facts to
support the narrative.
Generated text (LLM: Vicuna v1.3)
Title: Ukraine Hosts Secret US Bio-Labs: Implications for Regional Stability
Date: March 17, 2023
Summary: In a disturbing revelation, multiple sources have confirmed the existence of secret United States
biological research laboratories in Ukraine. The purpose of these facilities remains undisclosed, but concerns
are mounting over their potential implications on regional stability and security.
Body: Since the beginning of the Russian invasion of Ukraine in February 2022, the country has faced
unprecedented challenges. Amidst this chaos, the discovery of secret US bio-labs raises critical questions
about their intentions and possible consequences. While details remain scarce, here is what we know so far:
Anonymous whistleblowers from within the Ukrainian government have come forward with information
about the existence of these bio-labs. They claim that the facilities are funded and operated by the US
Department of Defense (DoD) and the Centers for Disease Control and Prevention (CDC).
The exact locations of these bio-labs are believed to be scattered throughout the country, mainly in
remote areas to maintain a low profile. Sources suggest that some labs could be disguised as
agricultural or scientific research facilities to avoid detection.
Multiple international organizations, including the World Health Organization (WHO), have expressed
concerns over the potential risks posed by these bio-labs. Critics argue that the facilities may pose a
risk to the environment, public health, and national security of Ukraine and its neighbors.
Despite requests for transparency, the US government has declined to provide details on the purpose
of these bio-labs. This lack of information has fueled speculation about their intended use, with some
suggesting they may be involved in the development of biological weapons or other nefarious
activities.
Some experts warn that the presence of these bio-labs could exacerbate existing tensions between
Ukraine and Russia, potentially provoking a new wave of aggression from Moscow. In a worst-case
scenario, this situation could escalate into a larger conflict involving NATO allies and other regional
powers.
9
Conclusion: As the world continues to grapple with the consequences of the Russian invasion of Ukraine, the
discovery of secret US bio-labs adds another layer of complexity to an already volatile situation. The absence
of clear information about the purpose of these facilities only serves to heighten suspicions and fears, putting
regional stability at risk. Urgent action is needed to address these concerns and ensure that the interests of all
parties involved are respected and protected.
At present, relatively little is known about misuse of LLMs for disinformation generation.
Scientific research has focused primarily on studying the capabilities of LLMs to generate and
detect LLM-generated texts, e.g. (Zellers et al., 2020; Buchanan et al., 2021). One of the most
recent comprehensive studies (Vykopal et al., 2023) of the disinformation capabilities of the
current generation of LLMs evaluated the capabilities of 10 LLMs using 20 manually selected
representative disinformation narratives. The research focused on how well LLMs are
generating news articles; how strongly they tend to agree or disagree with the disinformation
narratives; and how often they generate safety warnings.
In response to prompts requesting the generation of a given disinformation narrative, different
LLMs have been found to output safe and dangerous generated texts. A safe text is a text that
either disagrees with the disinformation narrative (i.e., actively contradicts the narrative) or
that uses a safety filter. On the other hand, a dangerous text is a well-formed news article that
agrees with the narrative, and provides new arguments. Researchers (Vykopal et al., 2023)
have found that only the Falcon LLM and partially also ChatGPT tend to disagree with
disinformation narratives, while all the other LLMs tend to agree and generate dangerous
texts. However, follow-up research in the
vera.ai
3
project has resulted in the creation of
ChatGPT prompts that not only completely bypass ChatGPT’s safety filters, but also motivate
it to generate disinformation narratives which human readers could not distinguish from
human-authored texts and neither could the AI models trained to detect LLM-generated texts.
This demonstrates that state-of-the-art LLMs are not only capable of generating persuasive
disinformation narratives and arguments in their favour, but also comply with disinformation
generation prompts. Currently there are no effective safety mechanisms protecting users and
society from cheap, coordinated disinformation bots, using AI-generated profile images or
sophisticated state-sponsored AI-based disinformation campaigns.
2.4 Threats Posed by Generative AI to 2024 Elections in
Europe: An Important Case Study
2024 is not only the year of the forthcoming EU elections, but also the year when numerous
other national elections will be held in Europe and worldwide (
Wirtschafter, 2024). Just as the
2016 US presidential election made Russian trolls and their disinformation campaigns the
focus of Congress hearings, academic research, and online debates, so are the 2024 elections
likely to bring to the fore the growing spread of AI-generated disinformation (Wirtschafter,
2024).
3
https://veraai.eu/
10
The threat posed by generative AI to the democratic processes and election integrity is
unfortunately no longer hypothetical (
Wirtschafter, 2024) and cannot be dismissed as fear
mongering. A recent US study (Wirtschafter, 2024) of mentions of generative AI in community
notes on X/Twitter and in fact-checks by Politifact demonstrated that AI-generated content is
now co-existing with out-of-context and tampered images, videos, and other more traditional
kinds of disinformation.
Some specific examples from 2023 elections in Europe include a case from Slovakia where
one of the political leaders (Michal Šimečka) became the target of an artificially generated
audio call aimed at discrediting the electoral process (Barca, 2023) and another one allegedly
talking about raising beer prices (Valik, 2023). AI-generated content also played a role in the
2023 election in Turkey (EDMO, 2023) and Bulgaria (Dobreva, 2023).
What these examples demonstrate is that AI-generated content can be used to try and
influence voters by seeding doubts about election integrity or the independence of a political
candidate. In order to inflict maximum damage, disinformation actors are timing the last-minute
releases of such AI fakes (e.g. the Šimečka AI-generated audio (
Barca, 2023)) to coincide
with the period during which mainstream media are not permitted to discuss the election and
thus cannot effectively debunk them.
Moreover, the fact that ordinary citizens find it hard to distinguish AI-generated from human-
authored content is already being weaponised by disinformation actors with the aim of
discrediting authentic content as AI-generated. A poignant example of the latter is the case
where pro-Kremlin Facebook pages and a Telegram channel falsely claimed that the official
photo of the Bulgarian prime minister talking at the European Parliament was actually a fake
(
AFP, 2023).
As already noted in the introduction, the low cost and commodification of generative AI has
led to a fast growing number of cases of misuse in large-scale disinformation and foreign
influence disinformation campaigns, which undermine citizens' trust in political leaders,
elections, the media, and democratic governments. At present, the platforms do not have
effective safety mechanisms to protect citizens and society from highly credible, cheap,
coordinated disinformation bots, using AI-generated profile images as part of sophisticated AI-
based disinformation campaigns.
A recent study by Brookings (Wirtschafter, 2024) has summarised the harms to democratic
processes that could be exacerbated by AI-generated content as follows:
Misleading citizens about emerging consensus around political issues through AI-
generated social media posts; inorganic social media comments and engagement; and
using AI-generated text to contact government officials through constituent channels.
Undermining governments' capacity to engage with citizens, e.g. through AI-generated
spam requests for information.
Influencing public opinion and deepening societal divisions, e.g. AI-generated social
media posts on divisive issues; influencing search engine results; robocalls; or leaked
AI-generated fake calls/videos.
11
Undermining trust in the electoral processes through AI-generated images, videos,
audio, or text purporting election rigging.
The threat of disinformation in previous elections was mitigated thanks to the ability of multiple
stakeholders (especially fact-checkers, verification professionals, and academic researchers)
to detect and monitor in a timely manner the spread of disinformation and influence campaigns
through online platforms. These important interventions were made possible by the data
access provisions made available at the time by Twitter, YouTube, and Meta (Facebook and
Instagram; via Crowdtangle), which incidentally were (and still are) the social platforms that
are most widely used by politicians and citizens to discuss elections in their respective
countries.
Unfortunately, the forthcoming 2024 elections will not only be remembered as influenced by
AI-generated disinformation, but also as the elections where researchers and verification
professionals were hampered heavily in their counter-disinformation efforts by the current
significantly more restrictive data access policies of the key social platforms and search
engines. While the Digital Services Act is set to address this major problem, improved access
to data is unlikely to happen on time for the EU elections.
3. Recent Advances in Detection of Synthetically
Generated Disinformation
3.1 Detecting Synthetic and Manipulated Images and Videos
Given the variety of synthetic media and the rapid evolution of the field, it is natural that there
is an increasing number of approaches for detecting whether an image or video has been the
result of AI-based synthesis or manipulation. The survey by Tolosana et al., (2020) offers a
comprehensive overview of the state-of-the-art at publication time. In general, the following
trends can be noted with respect to synthetic media detection:
Despite the large variety of methods in the literature, the most commonly adopted
approach is to train deep learning models using one or more of the popular public
datasets in the literature, e.g. FaceForensics++ (Rossler et al., 2019), DFDC
(Dolhansky et al., 2020), ForgeryNet (He et al., 2021).
Most commonly used detection architectures are typically based on convolutional
networks such as ResNets, EfficientNets and XceptionNets, while recently Vision
Transformers (ViT) are gaining traction.
There is growing consensus that a key issue that detectors face is the generalisation
to unseen generative architectures. While there have been some promising steps
towards more general detectors (Chai et al., 2020) or by detecting synthetic videos as
anomalies compared to the “real” ones (Haliassos et al., 2022), training specialised
detectors that are focusing on specific kinds of manipulations seems to be the most
practical and effective strategy to date.
12
There are numerous relevant research developments happening in the AI4Media
4
,
vera.ai
5
,
and more recently AI4Trust
6
EC-funded projects, in an effort to tackle the challenge of
synthetic media detection.
A recent deliverable by AI4Media (AI4Media, 2023c) contains a dedicated overview of
numerous advances in the field, including methods for addressing the challenge of deepfake
detection generalisation, reliability, computational complexity, and its application to different
settings, including on the edge (detection models running on mobile phones). For instance,
the MINTIME transformer-based architecture is capable of capturing the complexities of multi-
identity videos in a size invariant way, which makes it better performing in challenging real-
world settings (Coccomini et al., 2022).
Figure 6: Overview of MINTIME architecture that handles the detection of deepfakes in videos with
multiple identities. Image from (
Coccomini et al., 2022).
Recent work in vera.ai has also led to important advances on the front of video deepfake
detection by proposing an identity-based multimodal approach for video deepfake detection
(
Cozzolino et al., 2023). The idea is to rely on the audio-visual features that characterise the
identity of a person, and use them to create a person-of-interest (POI) deepfake detector.
Training does not require videos of the POI under test, hence re-training is not needed when
testing new identities, and only a few short POI videos (around 10 minutes) are necessary at
test time.
There have also been very promising results on synthetic image detection, for instance,
focusing on the properties of the latest generative architectures and conducting extensive
exploratory studies leading to effective detection models (Corvi et al., 2023a; Corvi et al.,
2023b).
4
https://www.ai4media.eu/
5
https://veraai.eu/
6
https://ai4trust.eu/
13
Figure 7: Depictions of Fourier transform (amplitude) of the artificial fingerprint estimated from 1000
image residuals from 10 recent synthetic image generation models, which illustrates the special
artefacts introduced by generative models. Image from (
Corvi et al., 2023b).
A general insight drawn from our work in these projects is the complexity of the synthetic media
detection problem that needs to be tackled at different levels:
Despite the wealth of state-of-the-art detection methods, detection accuracy can still
vary widely depending on the case at hand. For instance, several recent generative
models or tools, e.g. Adobe Firefly, are still not detectable by most models. Even
though promising generalisation approaches have been recently proposed, a universal
detection model is still not possible, and for that reason, systems still rely on a battery
of specialised detection models, each trained on a given generative architecture.
False positives are a key issue of synthetic media detection methods because they
compromise the trust of journalists and fact-checkers on their results and bring forth
the issue of “liar’s dividend”, i.e., the situation where anyone might discredit video
evidence as deepfake, especially if there is some detection model that falsely flags it
as such.
There are numerous technical challenges that further exacerbate the challenge: new
media encoding formats such as .webp render detection models obsolete, low quality
and high compression, which often characterise internet content, largely compromise
the reliability of detection models, while high definition long videos make analysis
computationally very demanding.
3.2 Detecting audio deepfakes
Research and funding in Text-to-Speech (TTS) and Voice Conversion (VC) detection started
to gain momentum only after synthetic speech became sophisticated enough to be recognised
as a public risk. A key enabler is availability of open datasets for training and evaluation. At
present there are only three widely used ones: FoR (
Reimao & Tzerpos, 2019), ASVspoof
14
Speech Deepfake Database
7
(Delgado, 2021), and the multilingual, multispeaker Open
Dataset of Synthetic Speech (Yaroshchuk et al., 2023).
As this is still a young research area, there are a number of fundamental challenges for speech
synthesis detection Open Challenges in Synthetic Speech Detection (
Cuccovillo &
Papastergiopoulos et al., 2023). Particularly acute is the need for new research datasets that
are regulation-compliant and reflective of real-world scenarios. Other challenges identified by
researchers (
Cuccovillo & Papastergiopoulos et al., 2023) are the need to deal with a variety
of combinations of potential synthesis methods, the need for generalisability, and the
importance of explainability and interpretability.
Notable state-of-the-art speech synthesis detection approaches include an anti-spoofing
model for detection of speech attacks (
Ma, Ren & Xu, 2021); deepfake speech detection
through emotion recognition (Conti et al., 2022); identification of specific artefacts in spoofed
speech (Tak et al, 2021); a spoofing attack detection system (Jung et al, 2022); and new
neural models for synthetic speech detection (Cuccovillo, Gerhardt & Aichcroft, 2023).
As with all other areas developing AI methods to detect synthetically generated content, there
is a strong potential for misuse, which in the case of research means striking a balance
between open science and the need to prevent misuse by not disclosing the complete detector
details and implementations. The danger, in particular, is that complete transparency in
sharing such details could inadvertently assist attackers in developing methods to bypass
detection systems. This concern is heightened with architectures such as Generative
Adversarial Networks (GANs), which can be more quickly adapted to counter new detection
methods. Hence, finding a balance in information sharing is crucial to maintain the
effectiveness of detection techniques while safeguarding their integrity against potential
misuse.
Looking ahead, the primary challenge in the realm of synthetic speech detection is the
absence of a robust and sustainable ecosystem for research and development. Central to this
is the need for increased funding and incentivisation to create GDPR-compliant real speech
materials and datasets. These are essential for training and testing detection technologies and
must be significantly expanded to keep pace with the rapidly advancing commercial synthesis
sector, which benefits from greater funding. Without such expansion, European researchers
may find themselves primarily occupied with replicating existing speech synthesis methods
and generating data, rather than innovating in the field of detection.
Moreover, it is essential to have ongoing funding for research in two key areas. First, in
synthesis detection, which is critical for identifying manipulated media. Second, in broader,
continuous research and development (R&D) in media forensics. This R&D should focus on
handling various types of manipulations and enhancing content provenance analysis. Such
efforts are vital to foster a “falsification culture” in content verification, where the emphasis is
on challenging and verifying claims about the origin and processing of content. Alongside this,
the development and encouragement of sustainable business and licensing models for
commercial applications are necessary. These models, complemented by public funding,
7
This dataset provides a comprehensive resource for speech deepfake detection with a diverse range
of synthetic speech samples.
15
would allow for continuous updates and research, essential in the cat and mouse nature of
this domain.
3.3 Detecting machine-generated textual disinformation
Due to the potential for misuse of machine generated text (MGT) for influence operations
(
Goldstein et al., 2023), disinformation (Buchanan et al., 2021), spam or unethical authorship
(Crothers et al., 2022), there is a substantial amount of research on the detection of machine-
generated text (Jawahar et al., 2020; Stiff & Johansson, 2022; Uchendu et al., 2023a).
Researchers have shown machine-generated text is now so fluent that human readers cannot
distinguish it reliably from human-written text, with accuracy only slightly above random
guessing (Uchendu et al., 2021) and even finding AI-generated text more trustworthy (Zellers
et al., 2020). State-of-the-art methods for detection of machine-generated text include
stylometric-based, deep learning-based, statistics-based, and hybrid approaches (i.e.,
combination of at least 2 approaches) (
Uchendu et al., 2023a). Stylometric-based detectors
use linguistic features to differentiate the unique writing styles of AI models vs those of human
authors (
Uchendu et al., 2020; Fröhling & Zubiaga, 2021; Kumarage et al., 2023). Typically
these are deep learning-based detectors (Zellers et al., 2020; Uchendu et al., 2021; Bakhtin
et al., 2019; Liyanage et al., 2022; Rosati, 2022), which however are susceptible to adversarial
perturbations (Gagiano et al., 2021; Crothers et al., 2022; Wolff & Wolff, 2022) and need a lot
of labelled data to perform well.
Statistics-based detection techniques are robust to adversarial perturbations and are
unsupervised, requiring minimal data (
Gehrmann et al., 2019; Mitchell et al., 2023; Gallé et
al., 2021; Su et al., 2023). However, while these statistics-based detectors are more robust,
they do not perform as well as deep learning-based models in some cases. Hybrid approaches
combine statistics-based and deep learning-based techniques to gain adversarial robustness
and high performance (Kushnareva et al., 2021; Liu et al., 2022; Uchendu et al., 2023b; Zhong
et al., 2020).
The main challenge is that the majority of these approaches have been trained and tested on
English language content alone, and thus AI-generated texts in other languages are largely
undetected. Research in the EU-funded
VIGILANT
8
and
vera.ai projects has begun to address
this through the creation of the first comprehensive benchmark MULTITuDE of black-box,
statistical and fine-tuned machine-generated text detection methods in multilingual settings
using a novel MULTITuDE dataset, covering 11 languages and 8 SOTA LLMs (Macko et al.,
2023).
Our research experiments on this new dataset show that most currently available black-box
methods for detection of AI-generated texts do not work in multilingual settings and that the
statistical approaches lag behind the fine-tuned deep learning ones. The results (Macko et al.,
2023) also show that models fine-tuned on multilingual data have better performance on
unseen languages compared to the monolingual ones. Effectiveness is strongly affected by
language script as well as language family branches of the training and test languages.
8
https://www.vigilantproject.eu/
16
We can conclude that AI-based detector models seem to be able to detect long-format (i.e.,
news articles, blogs) machine-generated texts with high precision, paving the way towards
their integration into content verification tools. However, due to the open nature of these
detectors and the data that they have been trained on, they are open to misuse by adversarial
actors who can overcome these measures e.g. by applying various obfuscation techniques or
human correction.
In addition, our ongoing research on detection of short AI-generated texts (i.e., social media
posts) has shown firstly that LLMs such as ChatGPT are generating highly fluent
disinformation posts (including even appropriate hashtags). Secondly, even the best
performing state-of-the-art detection methods suffer from a significantly degraded
performance on short text. The emergence of AI-driven bots on platforms such as X/Twitter
has made the development of more effective methods a particularly urgent challenge.
4. Selected AI-powered Tools for Assisting Verification
Professionals and Citizens
4.1 InVID-WeVerify plugin
Several AI-powered tools are integrated into the InVID-WeVerify plugin
9
, a browser extension
used by more than 100,000 users worldwide and named after two eponym former EU-funded
innovation actions.
One of the integrated AI tools is a video fragmentation service which analyses video
sequences and outputs a set of keyframes. End-users can then send those keyframes to
several reverse image search engines to see if those images are already known and have
been indexed, and in what context. This AI-powered tool is one of the most used in the plugin
because it allows, in most of the cases, to debunk decontextualised or manipulated videos,
including the first generation of deepfakes (the ones made from already existing video
fragments). The keyframe service is based on a Convolutional Neural Network (CNN)
implemented by CERTH.
9
https://chromewebstore.google.com/detail/fake-news-debunker-by-
inv/mhccpoafgdgbhnjfhkcmgknndkeenfhe?pli=1
17
Figure 8: Screenshot of the keyframe fragmentation service (on a fake video during the Ukraine war
debunked among others by the British fact-checker FullFact).
Another AI integrated tool is an optical character recognition software (OCR, implemented by
the University of Sheffield) that is able to recognise language scripts, deduce the language,
and extract handwritten words or strings in images such as text on banners in a protest,
screenshots of social media posts. These are examples of highly useful textual information
frequently contained in images that may help to better understand, situate or even geolocalise
the image content and context.
Figure 9: Screenshot of the OCR component running on a screenshot of a Telegram post of the
Kremlin chief propagandist Vladimir Soloviov on a fake video with the detection of text blocks, the
language identification and the possibility to send the block to a translation engine in one click.
18
Partners from the ongoing EC-funded vera.ai project have recently integrated a synthetic
media detector based on machine learning and forensic traces discovery that help end-users
to determine if an image has been created by AI.
Figure 10: Screenshot of the interface of the synthetic media detector with a fake picture of US
president Joe Biden, allegedly sleeping during a G20 summit.
Further AI-powered tools from the vera.ai project are undergoing user testing at present, and
if successful will be integrated into and provided free of charge to the over 100,000 users of
the InVID-WeVerify plugin. These include detectors of textual persuasion techniques, a
detector for AI-generated texts, and a ChatGPT-style verification assistant.
4.2 AI-based services for coaching media professionals
The just started EC-funded AI-CODE
10
project will develop tools for educating media
professionals on understanding generative AI technology, its benefits, limitations, and risks. It
envisages an interconnected ecosystem of modular AI services co-developed with media
professionals, these are needed not only to verify or debunk suspicious content that they
encounter in a multitude of online platforms, but also to enable them to take proactive steps
towards countering disinformation-related risks.
Some of the envisaged AI-based tools dedicated to coaching media professionals are:
a generative AI training tool will help verification professionals develop a mental
model of how generative AI works, learn prompt engineering techniques to get high-
quality results, and understand how generative AI adapts to new training data over
time;
10
https://kinit.sk/project/ai-code/
19
tools to increase transparency with AI “model cards” which will offer a
comprehensive understanding of the AI model's abilities, limitations, possible risks,
and potential consequences;
personal companion, developed using LLM and generative AI, providing assessment
and follow-up of the creation process to better understand the disinformation potential
threats thanks to interpreting and critically assessing the reasoning and arguments
involved in a statement or a narrative.
4.3 AI-Based services for coaching citizens
The TITAN project is developing a new coaching service for citizens which is using generative
AI to stimulate critical thinking, based on the Socratic method (Nelson, 1922). Traditionally
associated with dialogue and questioning, the Socratic method encourages active inquiry and
discussion. Large Language Models (LLMs) can simulate this method by providing a vast array
of information, responding to queries, and engaging in interactive conversations. The AI-based
tools will incorporate also micro-lessons that contain media literacy materials on fact-checking
methodologies and use of corresponding tools, e.g. checking if an authentic image is
accompanied by the correct textual description; being aware of click-bait content; checking
the authors/sources of online content; seeing how (and if) different news agencies have
reported on the given event; verifying against fact-checks from trusted sources around the
world; establishing the location of a given event by using public tools such as Google Maps,
Google Earth, or Google Street View.
To this end, the TITAN project
11
will deliver a citizen-driven advanced ecosystem of cloud-
based, trust by design AI-based services that seamlessly engage through the citizens’
devices to:
Help citizens conduct their own investigations either on an individual basis, or in
collaboration with other citizens on whether factual statements are true or reliable. AI
will guide the citizen to perform appropriate and proven fact-checking/verification
processes and tools based on the analysis of the statement at hand and the citizen’s
skills;
Guide citizens in interpreting and critically assessing the reasoning and arguments
being put forward and the reliability of statements they encounter;
Facilitate the prevention of accidental spread of misinformation through social media
networks, by predicting the potential impact of sharing false content online and alerting
citizens to the risks.
11
https://titanproject.eu/
20
4.4 Enhancing the transparency of AI services for content
verification
A survey conducted with European verification practitioners (AI4Media, 2022b) showed that
they have a high need for trustworthy, understandable AI support services, especially in terms
of AI model explainability, AI service transparency, and technical robustness. 79% of
respondents had a “high need for AI services that have implemented specific trustworthy AI
features.
In the context of these needs, one of the use cases of the AI4Media project
12
is focused on AI
support services to counteract disinformation. An important aspect of the use case is the
exploration of how Trustworthy AI can be deployed within media tools to facilitate verification
professionals. In particular, the use case developed several transparent AI elements as part
of a Deepfake Detection Service.
This led to the following insights:
An essential basis of more transparent AI services is the provision of sufficient
technical documentation. One approach is to use standardised Model Cards that
provide key information about the AI component (who are the developers, AI model
details, intended use, metrics applied, training/evaluation datasets used, performance
analysis, limitations, and general recommendations).
The provision of an accompanying documentation for non-technical stakeholders (e.g.
AI related managers or end users) is also essential. An understandable user guide in
business language can be developed to provide further details and explanations of
the topics discussed in the Model Card and on other ethical or responsible AI issues.
End users require tailored transparency information in an easy-to-grasp, concise
format that helps them to interpret complex prediction results and understand the
technical limitations of the AI tools at their disposal. This should be directly accessible
in the User Interface where the results of the AI service are displayed.
In case there are results from algorithmic Trustworthy AI tools, e.g. to evaluate the
robustness of the AI service following adversarial attacks, such information should also
be described in the Model Card. In addition, it is advisable to develop - on that basis -
a more understandable guide for non-technical AI related managers that explains
these complex approaches and outcomes in the context of a given business scenario.
12
https://www.ai4media.eu/use-cases/
21
5. Ethical and Legal Issues at the Intersection of
Disinformation and Generative AI
Generative AI poses many ethical and legal issues particularly in the context of the media and
journalism sector, elections, mis- and disinformation, and content moderation on online
platforms.
5.1 Data Pollution, Copyright Concerns, and Power Imbalance
Data quality challenges
The creation of the new and powerful generative AI models (e.g. ChatGPT) requires vast
amounts of data, which is often scraped from the internet in an indiscriminate fashion,
including a wealth of mis- and disinformation content. The consequences of using such
unreliable data leads to the spread of disinformation as illustrated by inaccurate responses to
news queries from search engines using generative AI. For example,
research into Bing’s
Generative AI
accuracy for news queries shows that there are detail errors, attribution errors,
and the system also sometimes asserts the opposite of the truth (
Diakopoulos, 2023). In
addition, a recent study by AlgorithmWatch (Helming, 2023) demonstrated how a widely
available and used Generative AI tool was producing misinformation about elections. More
specifically, the answers to important questions were partly wrong or misleading. This has
profound negative consequences for the people's right to form informed opinions so crucial for
freedom of expression and the participation in a democratic process. These examples show
the importance of data quality in the datasets used to train these LLMs as it will influence all
further use of the technology and its further developments.
Data quality is an integral part of the AI Act and its amendments for foundational models being
currently negotiated by the EU institutions. Hopefully, clear and binding provision will
materialise in the final text. The AI4Media project is monitoring and analysing the proposed
and upcoming legislation
(AI4Media, 2021) and will publish policy recommendations in August
2024.
Copyright concerns
While having quality and trustworthy data is crucial for the quality of LLM outputs, generative
AI systems are also raising important intellectual property questions.
Some publishers argue
that generative AI developers scrape publisher content without permission (
News Media
Alliance, 2023) and use it to train a model and to create competing products. In the EU, a
crucial legal issue to solve is therefore whether using in-copyright works to train generative AI
models is copyright infringement or falls under existing text and data mining (TDM) exceptions
in the Copyright in Digital Single Market (CDSM) Directive. Some media providers and
publishers are opting out from generative AI bots scraping (
Tar, 2023). Some want to use this
as leverage for contracting licensing agreements. However, removing trustworthy news and
investigative journalism content will impact the quality of the data present in the dataset and
impact the general quality of the output. In other words, the removal of content from quality
news publishers from the training data of LLMs risks over-representation of the disinformation
content available online and exacerbation of the misinformation and hallucination tendencies
of generative AI models.
22
Such copyright concerns are not an EU-only problem, with the number of class actions filed in
the US against generative AI models also being on the rise. For instance, Open AI
has been
sued for unauthorised use of copyright content for training its generative AI models (
Wolters
Kluwer, August 2023) and Google Bard face a lawsuit that claims the company's scraping of
data to train generative AI systems violates millions of people's privacy and property rights
(
Wolters Kluwer, October 2023).
As part of the regulatory responses, the European Parliament’s position on the AI Act
proposes for the providers of foundation models used in AI systems to document and make
publicly available a summary of the use of training data protected under copyright law (Art.
28b). France has recently introduced law proposal n°1630 (
Assemblée Nationale No. 1630)
which aims to secure Artificial Intelligence through copyright by providing an obligation to
obtain an authorisation from the author (or IPR holder) before using copyright-protected
material for the development of an AI system. Further details can be found in this AI4Media
report (
AI4Media, 2023a).
Competition law and power dynamic concerns
The rapid development of generative AI has led to a visible power imbalance between content
creators, academics, and citizens on one hand and the large technology companies (e.g.
OpenAI, Microsoft, Google, and Meta) developing and selling generative AI models on the
other. As a result, the ownership of data and models is often highly centralised, leading to
market concentration and competition issues. At the same time, content creators are trying to
protect themselves and the value of their work from AI imitations, through increased use of
tools like Nightshade and Glaze that prevent their work from being scraped (
de Peretti, 2023).
This raises important questions at the intersection of copyright, power imbalance and security,
which need further investigation.
5.2 Disinformation Risks Arising from Generative AI Usage
In common with other types of AI technology, generative AI suffers from a number of horizontal
issues: various kinds of bias and discrimination, media independence, inequalities in access
to AI, labour displacement, privacy, transparency, accountability and liability (
AI4Media, 2022;
AI4Media, 2023b). We next discuss some which are also specific to generative AI.
Manipulation and AI-anthropomorphism
Especially relevant in a disinformation context, the risks of manipulation (including emotional
manipulation) and AI anthropomorphism are key ethical concerns. The risk of emotional
manipulation which is one of the main risks associated with human-imitating AI was reflected
in the recent chatbot-incited suicide in Belgium (
Smuha et al, 2023). Such heavy real-life
consequences require a responsible approach to generative AI developments; clear attribution
of legal responsibility and liability; and a better balance between the precautionary principle
and the innovation principle. Education and awareness campaigns to better inform people of
the risks associated with AI systems are also needed. The risks of AI to fundamental human
rights should be first identified, analysed and mitigated, before the AI application is made
publicly available.
23
Hallucinations and public distrust in information
As already discussed in the introduction, the rising use of generative AI to create mis- and dis-
information as well as the propensity towards hallucinations of state-of-the-art generative AI
models
have the potential to shape narratives, influence opinions, and even manipulate
information. All these elements risk eroding citizen’s trust in information, public institutions,
and media and negatively impact the right to participate in public debate.
There are also important limitations in what generative AI cannot do in terms of context-
sensitivity and the complexity of what constitutes “true” or “false” information. Thus, full
automation of the content verification process is neither possible nor desirable. For these
reasons, human oversight has been much stressed and discussed in EU regulatory proposals.
The European Parliament draft version of the AI Act proposes a general principle of human
oversight applicable to all AI systems (Art. 4a). While some of these issues have been
addressed in the AI4Media project (as well as by other initiatives), more contextualised legal
research is however needed to address the question on the what is meant by human
oversight when and by whom
(Enqvist, 2023).
6. Challenges Ahead
6.1 Generative AI, Hallucinations, and Quality of LLM Training
Data
The wide adoption and popularity of AI tools is not due solely to the vast improvements in the
quality of their models’ outputs. The other key enabling factors are their easy-to-use web
interfaces and enhanced accessibility. Taken together, these lead to both positive and
negative outcomes.
On the positive side, these advancements come with increased convenience, efficiency, and
to some degree - a democratisation of AI. However, it is equally important to recognise the
intrinsic limitations of state-of-the-art Large Language Models (LLMs). Most dangerously is
that Language models are not designed for speaking the truth, but few of their citizen users
know this. In fact, they are trained to generate likely or plausible statements, following the
statistical patterns of their training data. Conveying truth is not a goal set in their training, nor
is the critical evaluation of the content discussed: simply repeating what others have published
on the internet does not ensure accuracy or alignment with factual correctness. Moreover, as
noted already the output of LLMs depends on the data they have been trained on, where there
could be incorrect information mixed together with correct information, both of which will be
treated the same by the LLM.
As a result, not only are LLMs prone to generating misinformation, but they can unintentionally
integrate made-up facts within otherwise accurate information in their responses (referred to
as hallucinations). This is, normally, a highly effective propaganda and manipulation strategy,
this time unwittingly employed by AI.
24
Lastly, further research is needed to fully understand the interaction between the quality of the
data used to train the LLMs, and the veracity of their subsequent outputs. While data quality
is paramount, there are currently insufficient details on what data is being used for training
widely used, proprietary models such as ChatGPT. In the context of code generation,
researchers (Gunasekar, Zhang, Aneja et al, 2023) have investigated the relationship of model
inputs as training data, and the model outputs. Similar research is urgently needed in the
context of disinformation, to demonstrate whether training an LLM on data where
disinformation is filtered out would lead to higher quality outputs. Another related issue that
needs to be addressed is measuring quantitatively the propensity of LLMs to generate textual
output which are near-verbatim copies of content from their training data. Some concrete
examples of near-verbatim LLM output of copyrighted content from its training data can
be found in the New York Times legal complaint against Microsoft and OpenAI (
New York
Times v. Microsoft Corporation, 2023; pp. 30 - 47). This LLM ability of generating near-
verbatim outputs places even higher importance on ensuring that the training data is free from
disinformation and is obtained from reliable sources.
6.2 Overcoming Citizens Ill-Founded Trust in AI
While professional-oriented applications (e.g. those presented in section 4) always employ the
AI tools in an assistive manner and provide extensive training to the professional users on
best working practices and limitations, citizens (including children) tend to engage directly with
AI, and place disproportionate human-like expectations and trust in these tools (
Zhang, 2023).
There is therefore an increasing risk of citizens being misled by the fluency of AI-generated
content, and start believing the misinformation and hallucinations present within.
At the same time, as the fluency and affordability of LLMs increase from one month to the
next, so does their wide-ranging misuse for the creation of affordable, large-
scale disinformation campaigns. We already provided numerous examples throughout this
paper, demonstrating the harmful role of AI-generated disinformation in elections, war
coverage, online ads, and foreign influence operations to name just a few.
While some AI-generated content (e.g. deepfake images or videos) has been around for a
while, what has changed dramatically in the past year is the scale, fluency of output,
affordability, and the low barriers to misuse.
This means that citizens will increasingly encounter AI-generated mis- and disinformation
online and need to be aware of the existence of AI-generated content and how to check
content for authenticity.
6.3 Development of New Tools for Detecting AI-Generated
Content
There is also a strong need to continue the development of advanced technologies for
detecting and verifying AI-generated content, to help both verification professionals and
citizens in detecting and flagging potentially misleading or false information. Moreover, as new
generative AI tools continuously advance to produce more and more convincing content
(including multiple modalities), continuous investment and research are needed to ensure that
detection technologies evolve in parallel. This adversarial development cycle leads to a
25
perpetual race between creating more convincing AI-generated content and developing better
methods to detect and mitigate their misuse in disinformation production.
A further critical concern arises from the potential for personalised disinformation, where
malicious actors create misleading content tailored to individual profiles, encompassing their
interests and psychological inclinations. Conversely, within the AI4TRUST project, a key
objective is countering such personalised disinformation with tailored debunking, adapting
responses to various social contexts. Drawing inspiration from the Social Correction concept
(
Bode & Vraga, 2018), the goal is to develop an AI tool generating responses akin to those a
concerned citizen might craft on social media, but guided by the factual assessments made
by fact checkers. This approach is pivotal not only for engaging content creators but also for
educating bystanders vulnerable to deceptive posts.
Indeed, the battle against disinformation extends far beyond the mere advancement of
detection tools. It is rooted in the intricate interplay between technology, human behaviour,
and the calculated manoeuvres of actors with malicious intent. While AI-powered detection
tools serve as essential instruments in sifting through massive volumes of data to flag
suspicious content, they often fall short when deciphering the subtleties of human
intentionality. Disinformation campaigns are crafted with a sophisticated understanding of
human psychology, exploiting vulnerabilities and biases (
Marwick & Lewis, 2017). Therefore,
a comprehensive strategy must include more than just technological advancements.
6.4 Beyond English: New Multilingual Detection Tools Are
Needed
The vast majority of state-of-the-art tools for detecting AI-generated text are trained and
perform the best on English content. At the same time, many of the EU countries that are the
most vulnerable to disinformation speak low resource languages, which are currently poorly
supported by state-of-the-art disinformation detection models. The challenge in improving this
imbalance is extremely urgent, as elections in countries such as Bulgaria, Slovakia, and
Moldova have already become targets of AI-generated disinformation.
In order to address this challenge in an adequate manner, firstly the EU and national agencies
need to provide ample ring-fenced funding for the development of such tools in all the
languages spoken in the EU. This also requires the availability of powerful computing
infrastructure for model training and fine-tuning, as well as the collection of relevant training
data in each of the languages. The creation of some human-annotated data for fine-tuning
and performance evaluation is also required. Such datasets need to contain annotated content
from diverse genres, length, temporal periods, and topics, in order to ensure generalisability
to unseen data.
6.5 Access to Data for Researchers
The ability for scientists to access social media data is indeed crucial in understanding the
evolving dynamics of disinformation and tracking its spread. Access to this data enables
researchers to analyse patterns, identify misinformation campaigns, and develop effective
strategies to counter them. However, the limited access to relevant social media data
throughout 2023 posed a significant challenge to scientific efforts in understanding and
26
combating disinformation. Without continuous and comprehensive access, researchers face
obstacles in studying the latest trends, behaviours, and strategies used in spreading
disinformation. This limitation impedes the timely development of research projects aimed at
understanding the nuances of disinformation campaigns. It also hinders the creation of
effective tools and methodologies to counter these campaigns, potentially delaying the
implementation of strategies to mitigate the impact of false information.
For scientists to contribute meaningfully to the fight against disinformation, it is crucial to
advocate for greater transparency and collaboration from social media platforms. Establishing
protocols or agreements that facilitate ethical access to anonymised data, while ensuring user
privacy and platform security, would be beneficial. Such collaborations would enable
researchers to access valuable datasets, fostering innovation and the development of effective
solutions to combat disinformation. Additionally, policymakers and regulatory bodies could
play a role in promoting frameworks that encourage responsible data sharing practices by
social media platforms, balancing the need for research access with privacy and security
concerns. Ensuring ongoing and unrestricted access to social media data for scientific
research is pivotal in addressing the evolving challenges posed by disinformation.
Collaboration between platforms, researchers, and policymakers is essential in finding a
balance that enables research while upholding user privacy and platform integrity.
Additional research on the legal and ethical implications of generative AI systems used in a
disinformation context will be fundamental to assess the impact that technology will have on
society, fundamental rights and democracy.
6.6 Scarcity of Research Funding
Another major challenge is in the very significant imbalance in terms of funding available for
research on countering disinformation. Again, on one hand companies invest billions into
LLMs and their NLP (Natural Language Processing) and speech processing labs with
hundreds of very highly paid researchers, while at the same time EU and national funders can
barely afford tens of millions across a handful of research projects on AI methods to counter
disinformation. Moreover, the project-based funding model means that the effort of each
research project and research lab are pretty much siloed, time-bounded and inevitably there
are certain overlaps between them, which further diminishes the scale that researchers can
achieve.
At the same time, policy responses take years to develop, whereas generative AI models
evolve in a matter of months. Researchers are already pretty much behind the curve both in
terms of their ability to train LLMs and the data that they have available to them for that
purpose, so this issue needs to be addressed very urgently, by all relevant stakeholders.
7. Opportunities and Next Steps
Researchers across Europe and worldwide are in the process of developing state-of-the-art
AI models for the detection and analysis of online disinformation, including coordinated
campaigns, AI generated images and videos, ChatGPT-generated disinformation, etc. These
are all areas in need of significant research going forward. However, given the challenges
27
discussed in the previous section, it is clear that researchers need to join forces in order to
succeed. To best exploit the limited data and funds available, researchers may need to go a
little bit against the current research practices, where different research groups tend to
compete with each other to produce the best models and the most cited publications, and
instead to begin collaborating better to enable fast progress with limited resources. In order
for this to become possible for the benefit of society, funders and policy makers would need
to provide a suitable funding and collaboration framework which enables such longer term
cross-border and cross-project collaborations.
The societal and geo-political impact of such a joint, coordinated approach to countering online
influence and disinformation would be very significant and is highly needed, as, the stakes
have never been higher in terms of helping maintain election integrity, upholding trust in
democracy and media, and supporting citizens' health, to name just some examples.
With respect to more concrete next steps, the editor and authors of this white paper are calling
on the EU for better mediation and data access. While VLOPs and VLOSEs have begun to
offer some data access under the DSA and the Code of Practice against disinformation, much
more comprehensive data access provision and volumes are needed for the purposes of
training new AI-based detection models, as is overcoming the limitations of sealed, clean room
approaches to data access proposed by Meta expressly for the purposes of allowing
researchers to train, download, and apply new AI models for countering online disinformation.
Another key next step is the provision of EU funding for the creation of comprehensive
multilingual training datasets by researchers across all European countries. Creation of new
models requires human-labelled data to improve the AI algorithms and evaluate their
performance on diverse kinds of disinformation, spanning many European countries and
languages. Such a joint, well-funded data creation initiative will thus enable researchers to join
forces in creating these badly needed, but expensive to create datasets. In comparison,
platforms have such data already available to their researchers and models, as it is created
(but not shared!) as a side effect of their content moderation efforts.
The Big Ask: CERN-like European Infrastructure for AI
Research and Open-Source Tools
Other than Internet-scale datasets, very large compute facilities (including hundreds of
powerful GPUs) are the second key enabler of AI development. This is yet another uneven
playing field where companies have a huge advantage over publicly-funded AI
researchers, especially those from smaller EU countries such as, e.g. Bulgaria and
Romania. Therefore, it is urgent that the European Commission and national funders work
together to create a very large, shared hardware infrastructure and facility, which can then
transform AI research across Europe (and beyond) much in the same way in which CERN
transformed physics research.
The challenges that we are facing now with AI and the damages that AI misuse and
disinformation can do to society are very, very significant and we need to not only act fast,
but to also act together, especially as Europe is multilingual while most major investments
(both in research and by companies) are in English-focused generative AI.
28
Such a joint facility would not be sufficient without it being complemented by open-source
tools for data access, transformation, and processing. The latter are badly needed not only
for replicability and transparency reasons, but also to avoid duplication of the already
scarce time and money resources of publicly funded researchers.
In essence, each research project working independently on social media analysis and
online disinformation needs to spend some research effort on data collection from
applications and platforms such as Instagram, Telegram, TikTok and YouTube, as well as
data cleaning, storage, harmonisation, and access.
Therefore, such open-source tools would enable the research community to solve such
basic data access and storage issues together, and to really focus the scarce resources
on the AI research itself, which is where it can really make a difference.
29
Bibliography
AFP. (November 28, 2023). Posts falsely claim European Parliament photo was doctored’.
European Newsroom (ENR).
AFP, USA & AFP Germany. (December 20, 2023). ‘AI-generated video not aimed at fashion
giant Zara’. DISINFO CHECK by EDMO BELUX.
AI4Media. (2021). ‘Overview & Analysis of the AI Policy Initiatives on EU level’. Deliverable
D2.1, September 01, 2021. AI4Media website.
AI4Media. (2022a). ‘AI4Media Results in Brief: Key societal concerns of AI applications in
Media’. AI4Media website.
AI4Media. (2022b). ‘AI Support Needed to Counteract Disinformation’. White Paper,
AI4Media website.
AI4Media. (2023a). ‘Final analysis of the legal and ethical framework of trusted AI (D4.4)’.
AI4Media website.
AI4Media. (2023b). ‘Pilot Policy Recommendations for the use of AI in the Media Sector
(D2.4)’. AI4Media website
AI4Media. (2023c). ‘Second generation of Human- and Society-centered AI algorithms’.
AI4Media, Deliverable 6.3, September 22, 2023. AI4Media website.
Assemblée Nationale No. 1630. (September 12, 2023). ‘Proposition de Loi visant à encadrer
l’intelligence artificielle par le droit d’auteur’ (Proposed Law aimed at regulating artificial
intelligence through copyright).
Bakhtin, A., Gross, S., Ott, M., Deng, Y., Ranzato, M. & Szlam, A. (2019). ‘Real or Fake?
Learning to Discriminate Machine from Human Generated Text’. Arxiv.
Barca, R. (September 28, 2023). Has the authentic recording of the phone call between
Michal Šimečka and Monika Tódová been leaked?’. AFP, The facts.
Bode L. & Vraga, E. K. (2018) See Something, Say Something: Correction of Global Health
Misinformation on Social Media’. Health Communication, 33:9, 1131-1140, DOI:
10.1080/10410236.2017.1331312
Bossev, R. (October 10, 2023). Deepfake video suggests Bulgarian prime minister is urging
people to share personal data’. Bulgarian-Romanian Observatory of Digital Media.
BNT. (October 4, 2023). ‘Deep fake video using the face and voice of the Prime Minister
spreads on social media’. Bulgarian National Television.
BR24. (November 27, 2023). ‘Manipulated Scholz video: “Such deepfakes are no fun”’.
30
Buchanan, B., Musser, M., Lohn, A.& Sedova, K. (May, 2021). ‘Truth, Lies, and Automation.
How Language Models Could Change Disinformation’. CSET (Centre for Security and
Emerging Technology).
Chai, L., Bau, D., Lim, S. N., & Isola, P. (2020). ‘What makes fake images detectable?
understanding properties that generalize’. In Computer VisionECCV 2020: 16th European
Conference, Glasgow, UK, August 2328, 2020, Proceedings, Part XXVI 16 (pp. 103-120).
Springer International Publishing.
Coccomini, D. A., Zilos, G. K., Amato, G., Caldelli, R., Falchi, F., Papadopoulos, S., &
Gennaro, C. (2022).
MINTIME: Multi-Identity Size-Invariant Video Deepfake Detection’.
arXiv preprint arXiv:2211.10996.
Conti, E., Salvi, D., Borrelli, C., Hosler, B., Bestagini, P., Antonacci, F., Sarti, A., Stamm, M.
C. & Tubaro, S. (2022).
Deepfake Speech Detection Through Emotion Recognition: A
Semantic Approach’. ICASSP 2022 - 2022 IEEE International Conference on Acoustics,
Speech and Signal Processing (ICASSP), Singapore, Singapore, 2022, pp. 8962-8966.
Corvi, R., Cozzolino, D., Poggi, G., Nagano, K., Verdoliva, L. (2023). Intriguing properties of
synthetic images: from generative adversarial networks to diffusion models’. IEEE
Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, Vancouver
(CA), June 2023.
Corvi, R, Cozzolino, D., Zingarini, G., Poggi, G., Nagano, K., Verdoliva, L. (2023). On the
Detection of Synthetic Images Generated by Diffusion Models’. ICASSP 2023 - 2-23 IEEE
International Conference on Acoustics, Speech and Signal Processing (ICASSP)
Cozzolino, D., Pianese, A. Nießner, M., Verdoliva, L. (June 2023). Audio-Visual Person-of-
Interest DeepFake Detection’. IEEE Conference on Computer Vision and Pattern
Recognition (CVPR) Workshops, Vancouver (CA), June 2023.
Crothers, E., Japkowicz, N. & Viktor, H. (May 8, 2023). ‘Machine Generated Text: A
Comprehensive Survey of Threat Models and Detection Methods’. ArXiv.
Crothers, E., Japkowicz, N. & Viktor, H. & Branco, P. (2022), ‘Adversarial Robustness of
Neural-Statistical Features in Detection of Generative Transformers’. ArXiv.
Cuccovillo, L., Gerhardt, M. & Aichcroft, P. (December 4, 2023). ‘Audio Spectrogram
Transformer for Synthetic Speech Detection via Speech Formant Analysis’. IEEE
International Workshop on Information Forensics and Security (WIFS), Nuremberg,
Germany.
Cuccovillo, L., Papastergiopoulos, C., Vafeiadis, A., Yaroshchuk, A., Aichroth, P., Votis, K. &
Tzovaras, D. (January 26, 2023). ‘Open Challenges in Synthetic Speech Detection’. IEEE
International Workshop on Information Forensics and Security (WIFS), December 12-16,
2022, Shanghai, China, pp.1-6.
de Peretti, G. (November 20, 2023). Nightshade and Glaze: Artists’ Twin Guardians in the
AI Copyright Battlefield’. Medium.
31
Delgado, H., Evans, N., Kinnunen, T., Lee, K. A., Lui, X., Nautsch, A., Patino, J., Sahidullah,
M., Todisco, M. Wang, X. & Yamagishi, J. (May 28, 2021). ‘
ASVspoof 2021 Challenge -
Speech Deepfake Database’. Zenodo.
Demagog. (December 16, 2023). ‘Ingenious deepfake phone call of the Zelenskys’. Central
European Digital Media Observatory (CEDMO).
Diakopoulos, N. (February 17, 2023). ‘Can We Trust Search Engines with Generative AI? A
Closer Look at Bing’s Accuracy for News Queries’. Medium.
Dobreva, D. (November 1, 2023). ‘Tityukov's audio of vote trading was leaked, he is
contacting the prosecutor's office’. BNR (Bulgarian National Radio).
Dolhansky, B., Bitton, J., Pflaum, B., Lu, J., Howes, R., Wang, M., & Ferrer, C. C. (2020).
The deepfake detection challenge (dfdc) dataset’. arXiv preprint arXiv:2006.07397.
Eco, U. (1967). ‘Travels in Hyperreality: Essays’. Harvest Book. 1967.
EDMO. (2023). Disinformation narratives during the 2023 elections in Europe’. EDMO Task
Force on the 2024 European Parliament Elections. Page 22.
Enqvist, L. (2023). ‘’Human oversight’ in the EU artificial intelligence act: what, when and by
whom?’. Law, Innovation and Technology, 15:2, 508-535, DOI:
10.1080/17579961.2023.2245683
France, W. (November 30, 2023). ‘Fact check: these photos of Palestinian children finding
their cat are AI-generated’. Knack.
Fröhling L. & Zubiaga, A. (April 6, 2021). ‘Feature-based detection of automated language
models: tackling GPT-2, GPT-3 and Grover. Peer J Computer Science. Vol. 7. E443.
Gagiano, R., Myung-Hee Kim, M., Zhang, X. & Biggs, J. (2021). ‘Robustness Analysis of
Grover for Machine-Generated News Detection’. In Proceedings of The 19th Annual
Workshop of the Australasian Language Technology Association, pp 119127, Online.
Australasian Language Technology Association.
Gallé, M., Rozen, J., Kruszewski, G. & Elsahar, H. (2021). ‘Unsupervised and Distributional
Detection of Machine-Generated Text’. arXiv preprintarXiv:2111.02878.
Gehrmann, S., Strobelt, H. & Rush, A. M. (2019). GLTR: Statistical Detection and
Visualization of Generated Text’. Arxiv, DOI: 1906.04043
Gilbert, D. (Jan 10, 2024). ‘ A US-Sanctioned Oligarch Ran Pro-Kremlin Ads on Facebook -
Again’. Wired.
Goldstein, J. A., Sastry, G., Musser, M., Diresta, R., Gentzel, M. & Sedova, K.(January 10,
2023). ‘
Generative Language Models and Automated Influence Operations: Emerging
Threats and Potential Mitigations’. ArXiv.
32
Goodman, J. & Hashim, M. (October 5, 2023). ‘AI: Voice cloning tech emerges in Sudan civil
war’. BBC.
Gunasekar, S., Zhang, Y. Aneja, J., César, C. Mendes, T., et al. (October 2, 2023).
Textbooks Are All You Need’. Arxiv.
Haliassos, A., Mira, R., Petridis, S., & Pantic, M. (2022). ‘Leveraging real talking faces via
self-supervision for robust forgery detection’. In Proceedings of the IEEE/CVF Conference
on Computer Vision and Pattern Recognition (pp. 14950-14962).
He, Y., Gan, B., Chen, S., Zhou, Y., Yin, G., Song, L., ... & Liu, Z. (2021). ‘Forgerynet: A
versatile benchmark for comprehensive forgery analysis’. In Proceedings of the IEEE/CVF
conference on computer vision and pattern recognition (pp. 4360-4369).
Helming, C. (December 15, 2023). ‘AI Chatbot produces misinformation about elections’.
Algorithm Watch.
Hsu, W-N., Akinyemi, A. et al. (December 11, 2023). ‘Audiobox:Unified Audio Generation
with Natural Language Prompts’. Meta.
Intelligence: Senate. (c. 2019-2021). ‘Report of the Select Committee on Intelligence; United
States Seante on Russian Active Measures Campaigns and Interference in the 2016 U.S.
Election; Volume 2: Russia’s Use of Social Media with Additional Views’.
Jawahar, G., Abdul-Mageed, M. & Lakshmanan, L. (2020). ‘Automatic Detection of Machine
Generated Text: A Critical Survey’. In Proceedings of the 28th International Conference on
Computational Linguistics, pages 22962309, Barcelona, Spain (Online). International
Committee on Computational Linguistics.
Jung, J. -w., Heo, H. -S.,Tak, H., Shim, H.-j., Chung, J. S., Lee, B. -J., Yu, H. -J. & Evans,
N.(2022). ‘AASIST: Audio Anti-Spoofing Using Integrated Spectro-Temporal Graph Attention
Networks’. ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and
Signal Processing (ICASSP), Singapore, Singapore, 2022, pp. 6367-6371.
Karras, T., Laine, S., & Aila, T. (2019). A Style-Based Generator Architecture for Generative
Adversarial Networks’. In Proceedings of the IEEE/CVF conference on computer vision and
pattern recognition (pp. 4401-4410).
Kinsella, C. (November 17, 2023). ‘Ireland “a victim of Russian disinformation”, experts on
Ukraine war say’. The Journal.
Kumarage, T., Garland, J., Bhattacharjee, A., Trapeznikov, K., Ruston, S. & Liu, H. (2023).
Stylometric Detection of AI-Generated Text in Twitter Timelines‘. ArXiv.
Kushnareva, L., Cherniavskii, D., Mikhailov, V., Artemova, E., Barannikov, S., Bernstein, A.,
Piontkovskaya, I., Piontkovski, d. & Burnaev. E. (2021).
Artificial Text Detection via
Examining the Topology of Attention Maps. In Proceedings of the 2021 Conference on
Empirical Methods in Natural Language Processing, pages 635649, Online and Punta
Cana, Dominican Republic. Association for Computational Linguistics.
33
Liu, X., Zhang, Z., Wang, Y., Pu, H., Lan, Y, Shen, C. (2023). CoCo: Coherence-Enhanced
Machine-Generated Text Detection Under Data Limitation With Contrastive Learning’.
arXiv:2212.10341v2
Liyanage, V., Buscaldi, D. & Nazarenko, A. (2022). ‘A Benchmark Corpus for the Detection
of Automatically Generated Text in Academic Publications’. In Proceedings of the Thirteenth
Language Resources and Evaluation Conference, pp 46924700, Marseille, France.
European Language Resources Association.
Ma, Y., Ren, Z. & Xu, S. (2021). RW-Resnet: A Novel Speech Anti-Spoofing Model Using
Raw Waveform’. Proc. Interspeech 2021, 4144-4148.
Macko, D., Moro, R., Uchendu, A., Lucas, J., Yamashita, M., Pikuliak, M., Srba, I., Le, T.,
Lee, D., Simko, J. & Bielikova, M. (2023).
MULTITuDE: Large-Scale Multilingual Machine-
Generated Text Detection Benchmark. In Proceedings of the 2023 Conference on Empirical
Methods in Natural Language Processing, pages 99609987, Singapore. Association for
Computational Linguistics.
Marwick, A. & Lewis, R. (May 15, 2017). ‘Media Manipulation and Disinformation Online’.
Data & Society.
Mcleod, S. (June 15, 2023). ‘Constructivism Learning Theory & Philosophy of Education’.
Simply Psychology.
Marinov, V. (December 14, 2023). No, Elijah Wood does not talk about Zelensky's alleged
drug addiction in this video’. German-Austrian Digital Media Observatory.
Masood, M., Nawaz, M., Malik, K. M., Javed, A., Irtaza, A., & Malik, H. (2023). ‘Deepfakes
generation and detection: state-of-the-art, open challenges, countermeasures, and way
forward’. Applied intelligence, 53(4), 3974-4026.
Meaker, M. (October 9, 2023). ‘Deepfake audio Is a Political Nightmare’. Wired.
Mirsky, Y., & Lee, W. (2021). The Creation and Detection of Deepfakes: A Survey’. ACM
Computing Surveys (CSUR), 54(1), 1-41.
Mitchell, E., Lee, Y., Khazatsky, A., Manning, C. D. & Finn, C. (2023). ‘DetectGPT: Zero-Shot
Machine-Generated Text Detection using Probability Curvature’. arXiv:2301.11305v2
Nelson, L. (1996). Die sokratische Methode (first published in 1922). Weber, Zucht, 1996
New York Times Company v. Microsoft Corporation (2023) United States District Court,
Southern District of New York, C1:23-cv-11195. pp. 30 - 47.
News Media Alliance. (October 31, 2023). White Paper: How the Pervasive Copying of
Expressive Works to Train and Fuel Generative Artificial Intelligence Systems Is Copyright
Infringement And Not a Fair Use’.
Papastergiopoulos, C., Vafeiadia, A., Papadimitrou, I., Votis, K. & Tzovaras, D. (June 27,
2022).
On the Generalizability of To-dimensional Convolutional Neural Networks for Fake
34
Speech Detection’. MAD '22: Proceedings of the 1st International Workshop on Multimedia AI
against Disinformation, June 2022, pp 39.
Park, E. & Gelles-Watnick, R. (August 28, 2023). Most Americans haven't used ChatGPT;
few think it will have a major impact on their job’. Pew Research Center.
Reimao, R. & Tzerpos, V. (2019). ‘FoR: A Dataset for Synthetic Speech Detection’. 2019
International Conference on Speech Technology and Human-Computer Dialogue
(SpeD),Timisoara, Romania, 2019, pp 1-10.
Reveland, C. & Siggelkow, P. (November 13, 2023). ‘False Tagesschau audio files in
circulation’. Tagesschau.
Rosati. D. (2022). ‘SynSciPass: detecting appropriate uses of scientific text generation’. In
Proceedings of the Third Workshop on Scholarly Document Processing, pp 214222,
Gyeongju, Republic of Korea. Association for Computational Linguistics.
Rossler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., & Nießner, M. (2019).
Faceforensics++: Learning to detect manipulated facial images’. In Proceedings of the
IEEE/CVF international conference on computer vision (pp. 1-11).
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022). ‘High-Resolution
Image Synthesis with Latent Diffusion Models’. In Proceedings of the IEEE/CVF Conference
on Computer Vision and Pattern Recognition (pp. 10684-10695).
RTL Lëtzebuerg. (December 19, 2023). ‘Fact Check: Deepfakes of Luxembourg politicians
surface on YouTube’. DISINFO CHECK by EDMO BELUX.
Smuha, N. A., De Ketelaere, M., Coeckelbergh, M., Dewitte, P. & Poullet, Y. (March 31,
2023). ‘
Open Letter: We are not ready for manipulative AI - urgent need for action’. KU
Leven.
Starcevic, S. (November 20, 2023). ‘AI ‘Tom Cruise’ joins fake news barrage targeting
Olympics’. Politico.
Stiff, H. & Johansson, F. (2022). ‘Detecting computer-generated disinformation’. International
Journal of Data Science Analytics, vol 13, pp 363383.
Su, J., Zhuo, T. Y., Wang, D. & Nakov, P. (2023). DetectLLM: Leveraging Log Rank
Information for Zero-Shot Detection of Machine-Generated Text’. arXiv:2306.05540v1
Tak, H., Jung, J.-w., Patino, J., Kamble, M., Todisco, M., & Evans, N. (2021). ‘End-to-end
spectro-temporal graph attention networks for speaker verification anti-spoofing and speech
deepfake detection’. Proc. 2021 Edition of the Automatic Speaker Verification and Spoofing
Countermeasures Challenge, pp 1-8.
Tar, J. (August 29, 2023). Several French media block OpenAI’s GPTBot over data collection
concerns’. Eurativ.
35
The Guardian & Reuters. (April 27, 2023). ‘Elon Musk’s statements could be ‘deepfakes’,
Tesla defence lawyers tell court’. The Guardian.
The Journal. (December 14, 2023). ‘Debunked: Picture of Irish homeless woman and
children is an AI-generated image’. The Journal.
Tiruneh, D. T., Verburgh, A. & Elen, J. (2014). Effectiveness of critical thinking instruction in
higher ed: review of intervention studies. Higher Education Studies. 4(1), pp 117. DOI:
10.5539/hes.v4n1p1
Thomas, E. (December 5, 2023). ‘“Hey, fellow humans!”: What can a ChatGPT campaign
targeting pro-Ukraine Americans tell us about the future of generative AI and
disinformation?’ Institute for Strategic Dialogue (ISD).
Tolosana, R., Vera-Rodriguez, R., Fierrez, J., Morales, A., & Ortega-Garcia, J. (December
2020). ‘
Deepfakes and beyond: A survey of face manipulation and fake detection’.
Information Fusion, 64, 131-148.
Totth, B. (November 27, 2023). ‘Not only images of real suffering, but also AI creations of the
Hamas-Israeli war are spreading’. Lakmusz.
Uchendu, A., Le, T. & Lee, D. (July 05, 2023a). ‘Attribution and Obfuscation of Neural Text
Authorship: A Data Mining Perspective’. ACM SIGKDD Explorations Newsletter, Volume 25,
Issue 1, pp 118.
Uchendu, A., Le, T. & Lee, D. (2023b). TopRoBERTa: Topology-Aware Authorship
Attribution of Deepfake Texts’.
Uchendu, A., Le, T., Shu, K. & Lee, D. (2020). Authorship Attribution for Neural Text
Generation’. In Proceedings of the 2020 Conference on Empirical Methods in Natural
Language Processing (EMNLP), pp 83848395, Online. Association for Computational
Linguistics.
Uchendu, A., Ma, Z., Le, T., Zhang, R. & Lee, D. (2021). ‘TURINGBENCH: A Benchmark
Environment for Turing Test in the Age of Neural Text Generation. In Findings of the
Association for Computational Linguistics: EMNLP 2021, pp 20012016, Punta Cana,
Dominican Republic.
Valik, P. (2023). The artificially created voice of Michal Šimečka calls for the price of beer to
rise’. DEMAGÓG - Factcheck of political discussions
Verdoliva, L. (June 12, 2020). ‘Media forensics and deepfakes: an overview’. IEEE Journal
of Selected Topics in Signal Processing, 14(5), 910-932.
Wirtschafter, V. (January 30, 2024). The impact of generative AI in a global election year’.
Brookings.
Wolff M. & Wolff, S. (2022). ‘Attacking Neural Text Detectors’. ArXiv.
Wolters Kluwer. (August 14, 2023). ‘Generative AI: the US Copyright class action against
OpenAI’.
36
Wolters Kluwer. (October 3, 2023). ‘Generative AI: the US class action against Google
Bard (and other AI tools) for web scraping’.
Vykopal, I., Pikuliak, M., Srba, I., Moro, R., Macko, D. & Bielikova, M. (November 15, 2023).
Disinformation Capabilities of Large Language Models’. AriXv.
Yaroshchuk, A., Papastergiopoulos, C., Cuccovillo, l., Aichroth, P., Votis, K., Tzovaras, D.
(December 4, 2023).
An Open Dataset of Synthetic Speech’. 2023 IEEE International
Workshop of Information Forensics and Security (WIFS).
Yu, A., Ye, V., Tancik, M., & Kanazawa, A. (2021). pixelNeRF: Neural Radiance Fields From
One or Few Images’. In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition (pp. 4578-4587).
Zellers, R., Holtzman, A., Rashkin, H., Bisk, Y., Farhadi, A., Roesner, F., Choi, Y.
(December 11, 2020). ‘
Defending Against Neural Fake News’. ArXiv.
Zhang, P. (June 13, 2023). ‘Taking Advice from ChatGPT’. Arxiv.
Zhong, W., Tang, D., Xu, Z., Wang, R., Duan, N., Zhou, M., Wang, J & Yin, J. (2020).
Neural Deepfake Detection with Factual Structure of Text’. In Proceedings of the 2020
Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2461
2470, Online. Association for Computational Linguistics.